1 Population parameters

1.1 Population-level causal questions

• Average causal effects for a population
  • e.g., What is the average effect of \(X\) on \(Y\)
• Proportion of different effects in a population
  • What share of cases in the population have positive effects?
  • What share have negative effects?
• Causal pathways
  • e.g., How commonly does \(X\) affect \(Y\) through \(M\) (vs. \(N\)) in the population?

1.2 Causal queries on a DAG: population-level average causal effect

• What is the average effect of \(X\) on \(Y\) in the population?
  • This is a question about the values in \(\lambda^Y\)

1.3 Causal queries on a DAG: population-level average causal effect

• With binary variables, the average effect is always the difference between
  • The share of cases with a positive effect
  • The share of cases with a negative effect
• \(\lambda_{01} - \lambda_{10}\)

1.4 Causal queries on a DAG: population-level average causal effect

\(ATE = \lambda^Y_{01} - \lambda^Y_{10}\)

1.5 ATE with a mediator

We need to

• Combine two ways of generating positive effects
• Combine two ways of generating negative effects
• Subtract the first from the second

1.6 ATE with a mediator

• Two ways of generating positive effects: \(\lambda^M_{01} \times \lambda^Y_{01}\) + \(\lambda^M_{10} \times \lambda^Y_{10}\)
• Two ways of generating negative effects: \(\lambda^M_{10} \times \lambda^Y_{01}\) + \(\lambda^M_{01} \times \lambda^Y_{10}\)
• Subtract the first from the second

1.7 Causal queries on a DAG: how often does \(X\) have a positive effect on \(Y\)?

• Proportion of cases with positive effect = \(\lambda^Y_{01}\)

1.8 Causal queries on a DAG: how often does \(X\) have a positive effect on \(Y\)?

• Proportion of cases with positive effect = \(\lambda^M_{01} \times \lambda^Y_{01}\) + \(\lambda^M_{10} \times \lambda^Y_{10}\)

1.9 Causal queries on a DAG: pathway questions

• We can also pose the pathway query at population level
  • What is the share of cases for which \(X\) has a positive effect on \(Y\) through \(M\)?
• A question about joint \(\lambda^M\) and \(\lambda^Y\) distributions

1.10 How will we answer questions about populations?

• We need to learn about those \(\lambda\)’s
  • About the proportions of the population with different kinds of causal effects
• We will have a prior belief about those proportions
• When we see data on lots of cases, we will update those beliefs about proportions
  • From a prior distribution over \(\lambda\) to a posterior distribution over \(\lambda\)

2 Intuition

2.1 How do we “update” our models?

• We’ve talked about process tracing a single case to answer a case-level query
  • Here the model is fixed
  • We use the model + case data to answer questions about the case
• We can also use data to “update” our models
  • Use data on many cases to learn about causal effects in the population
• Allows mixing methods: using data on lots of cases, we can learn about probative value of process-tracing evidence
• The core logic: we learn by updating population-level causal beliefs toward beliefs more consistent with the data

2.2 Start with a DAG

2.3 Large-\(N\) estimation of \(ATE\): what happens to beliefs over parameters

• We collect data on \(I\), \(M\), and \(D\) for a large number of cases
• We observe a strong positive correlation
• We will think there’s a positive average effect

2.4 Large-\(N\) estimation of \(ATE\): what happens to beliefs over parameters

• Now, we will update on both \(\lambda^I\) , \(\lambda^M\) and \(\lambda^D\)
• An \(I \rightarrow D\) effect can only happen if \(I\) affects \(M\) and \(M\) affects \(D\), in specific ways
• Two possible combinations of effects can generate a positive \(I \rightarrow D\) effect
  • \(I \rightarrow M\) is positive, \(M \rightarrow D\) is positive
  • \(I \rightarrow M\) is negative, \(M \rightarrow D\) is negative
• So we will come to put more weight on a joint distribution of \(\lambda^M\) and \(\lambda^D\) in which there are lots of cases with one of these two combinations
• …and less posterior weight on all other combinations of effects

2.5 General procedure

Key insight:

• If we suppose a given set of parameter values we can figure out the likelihood of the data given those values.
• We can do this for all possible parameter values and see which ones are more in line with the data

That, with priors, is enough to update:

\[p(\lambda | D) = \frac{p(D | \lambda)p(\lambda)}{p(D)}\] ## “By hand”

Let’s update manually.

2.6 In sum: learning from data

• For any data pattern, we gain confidence in parameter values more consistent with the data
• For single-case inference, we must bring background beliefs about population-level causal effects
• For multiple cases, we can learn about effects from the data
• Large-\(N\) data can thus provide probative value for small-\(N\) process-tracing
• All inference is conditional on the model

3 Mixed methods

3.1 A DAG

• We’ll want to learn about the \(\theta\)’s and the \(\lambda\)’s
• We need to observe nodes to learn about other nodes
• We can potentially observe 3 nodes here: \(X, M\), and \(Y\)

3.2 A typical “quantitative” data structure

• Data on exogenous variables and a key outcome for many cases

• E.g., data on inequality (\(I\)) and democracy (\(D\)) for many cases

3.3 A typical “qualitative” data structure

• Data on exogenous variables and a key outcome plus elements of process for a small number of cases
  • Finite resources mean tradeoffs between extensive and intensive data collection
• E.g., data on inequality (\(I\)), mass mobilization (\(M\)), and democracy (\(D\)) for many cases

3.4 Mixing qualitative and quantitative

• What if we combine extensive data on many cases with intensive data on a few cases?

• A non-rectangular data structure

3.5 Non-rectangular data

• A data structure that neither standard quantitative nor standard qualitative approaches can handle in a systematic way
• Not a problem for the Integrated Inferences approach
• We simply ask:
  • Which causal effects in the population are most and least consistent with the data pattern we observe?
• That is, what distribution of causal effects in the population, for each node, are most consistent with this data pattern?
• CausalQueries uses information wherever it finds it

3.6 Mixing in practice

For Bayesian approaches this mixing is not hard.

Critically though we maintain the assumption that cases for “in depth” analysis are chose at random—otherwise we have to account for selection processes.

What is the probability of seeing these two cases:

1. \(X=1, M = 1, Y = 1\) case
2. \(X=1, Y=1\) (no data on \(M\))

given parameters \(\lambda\):

3.7 Mixing in practice

The probability of 1 is:

\[p_{111}= \lambda^X_1 \times (\lambda^M_{01} + \lambda^M_{11}) \times (\lambda^Y_{01} +\lambda^Y_{11})\]

The probability of 2 is:

\[p_{1?1} = \lambda^X_1\times \left((\lambda^M_{01} + \lambda^M_{11}) \times (\lambda^Y_{01} +\lambda^Y_{11}) + (\lambda^M_{10} + \lambda^M_{00}) \times (\lambda^Y_{10} +\lambda^Y_{11}) \right)\]

So the probability of this data is just:

\[p(D|\lambda) = p_{111} * p{1?1}\]

3.8 Mixing in practice

Insight:

If we imagine possible parameter values we can figure out the likeihood of any data type – quantitative, qualitative or mixed.

That, with priors, is enough to update:

\[p(\lambda | D) = \frac{p(D | \lambda)p(\lambda)}{p(D)}= \frac{p(D | \lambda)p(\lambda)}{\int_{\lambda'}p(D|\lambda')p(\lambda')d\lambda'}\]

4 Why is this useful?

4.1 How qual can inform quant: confounding

Remember:

• Say we just observe a positive Inequality-Democratization correlation
• Could be because Inequality causes Democratization
• Could be because of confounding

4.2 How qual can inform quant: confounding

Remember

• Observing \(M\) helps
• Process data helps address the deep problem of confounding
• Key point: we don’t need \(M\) for all cases
  • Can learn from \(I\) and \(D\) for lots of cases and \(M\) for a subset

4.3 How qual can inform quant: observable confounder

• Another example: \(M\) as the confound

4.4 How qual can inform quant: observable confounder

• How much can we learn from \(M\) data for some cases?

4.5 How quant can inform qual: getting probative value of a clue from the data

• Suppose we go to the field and we learn that mass mobilization DID occur in Malawi

  • So \(M=1\)

• What can we conclude?

• NOTHING YET!

4.6 How quant can inform qual: getting probative value of a clue from the data

• The pure process-tracing solution: assign our beliefs about causal effects in the population
  • E.g., beliefs that linked positive effects are more likely than linked negative effects
  • Meaning that \(M=1\) in an \(I=1, D=1\) case speaks in favor of \(I=1\) causing \(D=1\)
• The mixed-methods solution: learn about population-level effects from large-\(N\) data

4.7 How quant can inform qual: getting probative value of a clue from the data

• Suppose we have data on \(I\), \(D\), and \(M\) for a large number of cases
• Suppose we observe a strong positive correlation across all 3 variables
• What have we learned, under this model?
  • Positive \(I \rightarrow M\) effects more likely than negative
  • Positive \(M \rightarrow D\) effects more likely than negative
• So linked positive effects more common than linked negative effects
• Meaning that \(M=1\) in an \(I=1, D=1\) case speaks in favor of \(I=1\) causing \(D=1\)
• But now we’ve now drawn our population-level beliefs from the data
• Now, we can go and process-trace
  • Did high inequality cause democratization in Malawi?
  • Observe \(M\)
• With conclusions grounded in case-level AND population-level evidence

5 Mixed methods in CausalQueries

5.1 Application

6 References